The apolipoprotein E (APOE) gene has been consistently shown to modulate the risk of Alzheimer’s disease (AD). Here, using an AD and normal aging dataset primarily consisting of three AD multi-center studies (n = 1,781), we compared the effect of APOE and amyloid-β (Aβ) on baseline hippocampal volumes in AD patients, mild cognitive impairment (MCI) subjects, and healthy controls. A large sample of healthy adolescents (n = 1,387) was also used to compare hippocampal volumes between APOE groups. Subjects had undergone a magnetic resonance imaging (MRI) scan and APOE genotyping. Hippocampal volumes were processed using FreeSurfer. In the AD and normal aging dataset, hippocampal comparisons were performed in each APOE group and in ɛ4 carriers with positron emission tomography (PET) Aβ who were dichotomized (Aβ+/Aβ–) using previous cut-offs. We found a linear reduction in hippocampal volumes with ɛ4 carriers possessing the smallest volumes, ɛ3 carriers possessing intermediate volumes, and ɛ2 carriers possessing the largest volumes. Moreover, AD and MCI ɛ4 carriers possessed the smallest hippocampal volumes and control ɛ2 carriers possessed the largest hippocampal volumes. Subjects with both APOE ɛ4 and Aβ positivity had the lowest hippocampal volumes when compared to Aβ- ɛ4 carriers, suggesting a synergistic relationship between APOE ɛ4 and Aβ. However, we found no hippocampal volume differences between APOE groups in healthy 14-year-old adolescents. Our findings suggest that the strongest neuroanatomic effect of APOE ɛ4 on the hippocampus is observed in AD and groups most at risk of developing the disease, whereas hippocampi of old and young healthy individuals remain unaffected.